Talk:Holy Warrior's Challenge/@comment-30176423-20161116032002/@comment-28059753-20161117101908
@Ssvb *"Just read the comments for the past events ... All the information is there." As I explained above, I don't take comments on event pages seriously as statistical evidence due to sampling bias. *"One of my posts was even a direct reply to you about getting 10 Lucille drops in 11 sorties." Granted. I'll accept that this data is good, but as I see it, this doesn't really have anything to do with the issue of your model of different people having different drop rates, or external factors having some influence on the drop rate. For any evidence of that position, you would need to exhibit different people with different drop rates or one person seeing different drop rates under different conditions. (In fact, to the contrary, user Yoxall reports getting "lucky" on the same map.) This result is perfectly explainable in the model of a fixed drop rate, just with a drop rate higher than 50%. I agree it's likely that the drop rate was indeed higher than 50%. (Without any data about this event in particular, I don't see any problem with people guessing that the drop rate would be 50% based on evidence of past events or whatever as long as they make it clear that they're guessing/speculating. Generally, it seems they do mention this caveat if my recollection serves.) *"For example, have a look here" Again, this is unsolicited data, so I'm immediately skeptical due to the possibility of sampling bias. We don't even conclusively know how many runs were made. But, let's take that as given. Then, again, this is not evidence in favor of your model, but simply evidence against the 8% drop rate. I agree that the drop rate is almost certainly much higher, (and, indeed, the average among results reported to the drop rate initiative was ~16%, but still with quite a wide interval estimate, so I would not be surprised at all if 16% is still low). *"Do you understand that *all* results have been reported and nothing filtered out?" Yes, all *your* results were reported, and indeed all the results of each individual person who reported results (perhaps), but not all the results of every user -- the vast majority of people didn't report any results. I don't think anyone can reasonably dispute that some people will be more likely post to an event page about results if they are extreme (very high or very low compared to the "advertised" or expected rate). This is sampling bias if you try to use such data. The data will be disproportionate in favor of these extreme results. But even if I am wrong, and these extreme results are less likely to be reported, there is sampling bias the other way. The only way there is no sampling bias is if **everyone's** decision whether to share their data is exactly statistically independent from the data they are sharing. To make such a suggestion is, frankly, ridiculous to me. I'll admit to not being an expert in statistics (though I'm certainly familiar with the binomial distribution, statistical power, p-value, etc) and not having done the numbers myself in regards to the effect that sampling bias may have, but if you don't think it's serious, and would like a moderately accessible (to a student of mathematics) overview, I might suggest watching this (a talk I quite enjoyed when I saw it at the JMM). *"As for the Belinda event..." Thank you for reiterating the specific data you're talking about. As I already have said regarding the Belinda event, I agree that the 8% number is almost certainly wrong. If we take, say 24%, as the drop rate (the upper end of the interval estimate from the drop rate initiative), then the chance of 10/20 or better is ~1% which is still unlikely, but ultimately believable. (Although, I should reject this interval in light of your data since the interval is only 95% credible -- but I don't believe that this post of yours is particularly free from sampling bias since on the wiki were apparently regarding drop rates during this event and it additionally doesn't record the exact numbers during this time period.) Once again, of course, you are debating against the 8%. I thought this thread was arguing against the constant drop rate model, not against specific numbers. *"And even if we assume that the drop rate is constant but just something higher than 8%, then we also have people who had reported zero drops in 21 and 25 runs." Now, this is actually evidence in favor of your position. Although, I'll preface this, of course, by repeating that I don't think this data can seriously be used for statistical inference due to sampling bias. I looked through most of two pages of the comments on The Witch Girl, and I didn't see these reports. The closest I saw was this comment which could be interpreted as 23 runs without Belinda, but 12 of those runs were on a different mission, and it's not clear to that Belinda is the subject of discussion anyway until the end of the comment which is even more vague. So, I guess you were talking about something else that I didn't see (quite possible), or you were mistaken. I'll also mention in passing that the 8% number was always wrong. The cited "108" from aigis.dougax.net (nsfw site warning) is actually supposed to be interpreted as 9% (as mentioned elsewhere on the site -- I don't have a reference handy, but I'll try to dig it up later if someone cares). This isn't particularly relevant to the present conversation though. I don't think it's bad, though, for people to offer the data from a previous incarnation of the event as a speculation in the absence of better data -- presuming that it is made clear that the number is just speculation. (Again, this generally does seem to be the case.) (I just wish we could get the number right.) *"Feedback is welcome." Your theory (or at least arguments) seem partially based on rejection of the drop rates that have been advertised on the event pages rather than evidence against the theory of fixed drop rates in general. Besides the data you have collected personally, you are basing your conclusions on data that I consider to be affected by sampling bias. I can't at the moment provide any specific argument that your theory is wrong, and I think it would be very hard to do so. But, there are substantial factors that make me believe that the fixed drop rate model, in absence of any hard evidence, is far more likely: *Your model is complicated. Why would the game developers implement such a model when a simpler model would work just as well? (If it was unintentional, then I think they would have made an enormous mistake.) *If the game uses your model, players would be able to exploit the model to improve their drop chances. Why would the developers implement such a system? *There is evidence based on numbers found in old builds of the game (already mentioned above) that the game previously used a simple model of fixed drop rates. Why would the developers have changed it? Although, obviously, none of these things is conclusive, these are the kinds of arguments you are unavoidably working against when you put forward a model like yours. If you're serious about providing evidence for your theory, I would suggest you need to collect more data. One straightforward test would be to collect a string of results under conditions that you don't believe will change the drop rate. Then perform an action that you believe will change the drop rate and collect another string of results. Try to show a statistically significant difference between these drop rates. (Of course, they are expected to be a little different even under a fixed drop rate model.) This may seem unappealing since you would be intentionally forgoing your chance at maximizing your drop chances on your own account (assuming your model is correct). You might offset this by running multiple accounts (which, by the way is against Nutaku's Terms of Use, so don't say I didn't warn you), but this will probably be very time consuming. In any case, you will probably want to solicit data from other users under the condition that the use practices that will eliminate or minimize sampling bias under your theory (something I commented on above). Good luck on getting people to help you. I have no suggestion except to learn from my similar efforts (for better or worse). No, I won't help you collect data; I have too much to do as it is. However, if you like, I will make available to you my raw drop data which at least includes my results (for those runs I recorded) in order on a run-by-run basis. I also have, on the backburner, an automated drop recording system (plugin for http://www.telerik.com/fiddler) I am working on which could be of use to you at some point. I haven't been designing it with your model in mind, of course, but you might could attempt something similar, or even go along the lines of just recording all traffic if you don't know which factors might be relevant. Another method may be to attempt to exploit your proposed model to maximize your own drops. In this case, if your results are better than the results of other people who are not following your method by a statistically significant margin, then you may have some evidence. I don't think this test is as powerful, but it's probably what you want to do anyway. Ideally, you would want to test against people who are trying to use your method to minimize their results, but as long as there is no sampling bias of these results under the model you are trying to disprove, you are probably okay. (E.g., I think you could compare against results collected by the drop rate initiative.) I really don't have any issue with you proposing or talking about your theory. The main reason I entered this conversation is because of you suggested you had specific data about this model but you didn't share it specifically. P.S. To other readers: I don't mind if you report your results on the event pages. My only issue comes when somebody then tries to use your data for statistical inferences. If you made no claims to the statistical validity of your data, the onus is not on you. Feel free to continue as usual.